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Joil Jaall 
“The AI Revolution: Our Immortality or Extinction?” Wait But Why (January27, 2015), at 
http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2 .htmil. 


This open letter, “Research Priorities for Robust and Beneficial Artificial Intelligence,” can 
be found at http://futureoflife.org/ai-open-letter/. 

Example of classic robot alarmism in the media: Ellie Zolfagharifard, “Artificial Intelligence 
‘Could Be the Worst Thing to Happen to Humanity,'” Daily Mail, May 2, 2014; http:/Ainyurl. 
com/hawkingbots. 


SU Jadll 
Notes on the origin of the term AG: http://wp.goertzel.org/who-coined-the -term-agji. 


Hans Moravec, “When Will Computer Hardware Match the Human Brain?” Journal of 
Evolution and Technology (1998), vol. 1. 


In the figure showing computing power versus year, the pre-2011 data is from Ray Kurzweil's 
book How to Create a Mind, and subsequent data is computed from the references in 
https://en.wikipedia.org/wiki/FLOPS. 


Quantum computing pioneer David Deutsch describes how he views quantum computation 
as evidence of parallel universes in his The Fabric of Reality: The Science of Parallel 
Universes —and Its Implications (London: Allen Lane, 1997). 1f you want my own take on 
quantum parallel universes as the third of four multiverse levels, you'll find it in my previous 
book: Max Tegmark, Our Mathematical Universe: My Quest for the Ultimate Nature of 
Reality (New York: Knopf, 2014). 


SJ Jadll 
Watch “Google DeepMind's Deep Q-learning Playing Atari Breakout” on YouTube at htips:// 
tinyurl.com/atariai. 

See Volodymyr Mnih et al., “Human-Level Control Through Deep Reinforcement Learning,” 
Nature 518 (February 26, 2015): 529-533, available online at http:/tinyurl.com/ataripaper. 
Here's a video of the Big Dog robot in action: https://www.youtube.com/watch?v=W1czBcnX1Ww. 
For reactions to the sensationally creative line 5 move by AlphaGO, see “Move 371! Lee 
Sedol vs AlphaGo Match 2," at https://www.youtube.com/ watch?v=JNrXgpSEEIE. 
Demis Hassabis describing reactions to AlphaGo from human Go players: https://www. 
youtube.com/watch?v=ot/KzpNWZTA4. 

For recent improvements in machine translation, see Gideon Lewis-Kraus, “The Great A.I. 
Awakening,” New York Times Magazine (December 14, 2016), available online at http:// 
www.nytimes.com/2016/12/14/magazine/the-great-ai -awakening.html. Google Translate 
is available here at https://translate.google.com. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 
ai. 
22. 


23. 


24. 


25. 


26. 


27. 


28. 


29. 
30. 


31. 
92. 


3.06U | 312 


Winograd Schema Challenge competition: http:/tinyurl.com/winogradchal lenge. 
Ariane 5 explosion video: https://www.youtube.com/watch?v=qnHn8W1Em6E. 
Ariane 5 Flight 501 Failure report by the inquiry board: http:/inyurl.com/ arianeflop. 


. NASA's Mars Climate Orbiter Mishap Investigation Board Phase | report: http:/Ainyuri.com/marsilop. 
. The most detailed and consistent account of what caused the Mariner 1 Venus mission 


failure was incorrect hand-transcription of a single mathematical symbol (a missing overbar): 
http://tinyurl.com/marinertlop. 


A detailed description of the failure of the Soviet Phobos 1 Mars mission can be found in 
Wesley T. Huntress Jr. and Mikhaïl Ya. Marov, Soviet Robots in the Solar System (New 
York: Praxis Publishing, 2011), p. 308. 


How unverified software cost Knight Capital $440 million in 45 minutes: http://tinyurl.com/ 
knightflop1i and http://tinyurl.com/knightflop2. 


U.S. government report on the Wall Street “flash crash": “Findings Regarding the Market 
Events of May 6, 2010” (September 30, 2010), at http:/tinyurl.com/ flashcrashreport. 


3-D printing of buildings (https://www.youtube.com/watch?v=SObzNdyRTBs), micromechanical 
devices (http:/inyurl.com/inyprinter) and many things in between (https://www. youtube. 
com/watch?v=xVU4FLrsPXSs). 


Global map of community-based fab labs: https://www.fablabs.io/labs/map. 


News article about Robert Williams being killed by an industrial robot: http:// tinyurl.com/ 
williamsaccident. 


News article about Kenji Urada being killed by an industrial robot: http:// tinyurl.com/ 
uradaaccident. 


News article about Volkswagen worker being killed by an industrial robot: http:/tinyurl. 
com/baunatalaccident. 


U.S. government report on worker fatalities: https://www.osha.gov/dep/atcat/ dep_fatcat.htmil. 
Car accident fatality statistics: http:/tinyurl.com/roadsafety2 and http:/Ainyurl com/roadsafety3. 


On the first Tesla autopilot fatality, see Andrew Buncombe, “Tesla Crash: Driver Who Died 
While on Autopilot Mode ‘Was Watching Harry Potter,” Independent (July 1, 2016), http:// 
tinyurl.com/teslacrashstory. For the report of the Office of Defects Investigation of the U.S. 
National Highway Traffic Safety Administration, see http://tinyurl.com/teslacrashreport. 


For more about the Herald of Free Enterprise disaster, see R. B. Whittingham, The Blame 
Machine: Why Human Error Causes Accidents (Oxford, UK: Elsevier, 2004). 


Documentary about the Air France 447 crash: htips:/{www. youtube.com/watch?v=dpPkpBOGQF; 
accident report: http://tinyurl.com/af447report; outside analysis: http://tinyurl.com/ 
thomsonarticle. 


Official report on the 2003 U.S.-Canadian blackout: http:/tinyurl.com/ uscanadablackout. 


Final report of the President's Commission on the Accident at Three Mile Island: http:// 
www.threemileisland.org/downloads/188.pdf. 


Dutch study showing how Al can rival human radiologists at MRI-based diagnosis of 
prostate cancer: http:/tinyurl.com/prostate-ai. 


Stanford study showing how Al can best human pathologists at lung cancer diagnosis: 
http:/Minyurl.com/lungcancer-ai. 


Investigation of the Therac-25 radiation therapy accidents: http://tinyurl.com/theractfailure. 


Report on lethal radiation overdoses in Panama caused by confusing user interface: http:// 
tinyurl.com/cobalt60accident. 


Study of adverse events in robotic surgery: htips://anrxiv.org/abs/1507.03518. 
Article on number of deaths from bad hospital care: http:/tinyurl.com/medac cidents. 
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33. 


34. 


35. 


36. 


37. 


38. 


39. 


51. 


53. 
54. 


Yahoo set a new standard for “big hack" when announcing a billion(!) of their user accounts 
had been breached: https://www.wired.com/2016/12/yahoo-hack -billion-users/. 


New York Times article on acquittal and later conviction of KKK murderer: http://tinyurl. 
com/kkkacquittal. 


The Danziger et al. 2011 study (http://www.pnas.org/content/108/17/6889 full), arguing 
that hungry judges are harsher, was criticized as flawed by Keren Weinshall-Margela and 
John Shapard (http://www.pnas.org/content/108/42/ E838.full), but Danziger et al. insist 
that their claims remain valid (http://www .pnas.org/content/108/42/E834.full). 


Pro Publica report on racial bias in recidivism-prediction software: http:/inyurl .com/ 
robojudge. 

Use of fMRI and other brain-scanning techniques as evidence in trials is highly controversial, 
as is the reliability of such techniques, although many teams claim accuracies better than 
90%: http://journal.frontiersin.org/article/10.3389/fpsyg .2015.00709/full. 

PBS made the movie The Man Who Saved the World about the incident where Vasili 
Arkhipov single-handedly prevented a Soviet nuclear strike: https://www .youtube.com/ 
watch?v=4VPY2SgyG5w. 

The story of how Stanislav Petrov dismissed warnings of a U.S. nuclear attack as a false 
alarm was turned into the movie The Man Who Saved the World (not to be confused with the 
movie by the same title in the previous note), and Petrov was honored at the United Nations 
and given the World Citizen Award: https:// www. youtube.com/watch?v=IncSjwWQHMo. 


. Open letter from Al and robotics researchers about autonomous weapons: http://futureotlite. 


org/open-letter-autonomous-weapons/. 


. A US. official seemingly wanting a military Al arms race: http:/tinyurl.com/ workquote. 
. Study of wealth inequality in the United States since 1913: http://gabriel -zucman.eu/files/ 


SaezZucman2015.pdf. 


. Oxfam report on global wealth inequality: http:/tinyurl.com/oxfam2017. 
. For a great introduction to the hypothesis of technology-driven inequality, see Erik Brynjolfsson 


and Andrew McAfee, The Second Machine Age: Work, Progress, and Prosperity in a Time 
of Brilliant Technologies (New York: Norton, 2014). 


. Article in The Atlantic about falling wages for the less educated: http:/tinyurl .com/wagedrop. 
. The data plotted are taken from Facundo Alvaredo, Anthony B. Atkinson, Thomas Piketty, 


Emmanuel Saez and Gabriel Zucman, The World Wealth and Income Database (htip:// 
www.wid.world), including capital gains. 


. Presentation by James Manyika showing income shifting from labor to capital: http:// 


futureoflife.org/data/PDF/james_manyika.pdf. 


. Forecasts about future job automation from Oxford University (http:// tinyurl.com/ 


automationoxford) and McKinsey (http://tinyurl.com/automation mckinsey). 


. Video of robotic chef: https://www.youtube.com/watch?v=fE6i2006Y6s. 
50. 


Marin Soljabiü explored these options at the 2016 workshop Computers Gone Wild: impact 
and Implications of Developments in Artificial Intelligence on Society: http://futureoflife. 
org/2016/05/06/computers-gone-wild/. 

Andrew McAfee's suggestions for how to create more good jobs: http://future oflife.org/ 
data/PDF/andrew_mcafee.pdf. 52. In addition to many academic articles arguing that “this 
time is different” for technological unemployment, the video “Humans Need Not Apply” 
succinctly makes the same point: https:/www. youtube.com/watch?v=7Pq-S557XQU. 
U.S. Bureau of Labor Statistics: http://www.bls.gov/cps/cpsaat11.htm. 


Argument that “this time is different” for technological unemployment: Federico Pistono, 
Robots Will Steal Your Job, but That's OK (2012), http://robotswill stealyourjob.com. 


55. 
56. 


57. 


58. 


59. 
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Changes in the U.S. horse population: http://tinyurl.com/horsedecline. 


Meta-analysis showing how unemployment affects well-being: Maike Luhmann et ai. 
“Subjective Well-Being and Adaptation to Life Events: À Meta-Analysis,” Journal of Personality 
and Social Psychology 102, no. 3 (2012): 592; available online at https://www.ncbi.nim. 
nih.gov/pmc/articles/PMC3289759. 

Studies of what boosts people's sense of well-being: Angela Duckworth, Tracy Steen and 
Martin Seligman, “Positive Psychology in Clinical Practice,” Annual Review of Clinical 
Psychology 1 (2005): 629-651, online at http:/tinyurl.com/well beingduckworth. Weiting Ng 
and Ed Diener, “What Matters to the Rich and the Poor? Subjective Well-Being, Financial 
Satisfaction, and Postmaterialist Needs Across the World,” Journal of Personality and Social 
Psychology 107, no. 2 (2014): 326, online at http://psycnet.apa.org/journals/psp/107/2/326. 
Kirsten Weir, “More than Job Satisfaction," Monitor on Psychology 44, no. 11 (December 
2013), online at http://www.apa.org/monitor/2013/12/job-satisfaction.aspx. 

Multiplying together about 10" neurons, about 104 connections per neuron and about one (10°) 
firing per neuron each second might suggest that about 105 FLOPS (1 petaFLOPS) suffice to 
simulate a human brain, but there are many poorly understood complications, including the 
detailed timing of firings and the question of whether small parts of neurons and synapses 
need to be simulated too. IBM computer scientist Dharmendra Modha has estimated that 38 
petaFLOPS are required (http://tinyurl.com/javin43), while neuroscientist Henry Markram has 
estimated that one needs about 1,000 petaFLOPS (http://tinyurl.com/6rpohqv). Al researchers 
Katja Grace and Paul Christiano have argued that the most costly aspect of brain simulation 
is not computation but communication, and that this too is a task in the ballpark of what the 
best current supercomputers can do: http://aiimpacts.org/about. 


For an interesting estimate of the computational power of the human brain: Hans Moravec 
“When Will Computer Hardware Match the Human Brain?” Journal of Evolution and 
Technology, vol. 1 (1998). 


gli Laall 
For a video of the first mechanical bird, see Markus Fischer, “A Robot That Flies like a 
Bird,” TED Talk, July 2011, at https://www.ted.com/talks/a_robot_ that_flies_like_a_bird. 


QuelsJl Laall 

Ray Kurzweil, The Singularity Is Near (New York: Viking Press, 2005). 

Ben Goertzel's “Nanny Al” scenario is described here: https://wiki.lesswrong .com/wiki/ 
Nanny_Al. 

For a discussion about the relationship between machines and humans, and whether 
machines are our slaves, see Benjamin Wallace-Wells, “Boyhood,” New York magazine 
(May 20, 2015), online at http///tinyurl.com/aislaves. 

Mind crime is discussed in Nick Bostrom's book Superintelligence and in more technical detail in this 
recent paper: Nick Bostrom, Allan Dafoe and Carrick Flynn, "Policy Desiderata in the Development 
of Machine Superintelligence" (2016), http:/www.nickbostrom.com/papers/aipolicy.pdf. 

Matthew Schofield, “Memories of Stasi Color Germans’ View of U.S. Surveillance 
Programs,"McCiatchy DC Bureau (June 26, 2013), online at http://www .mcclatchyde. 
com/news/nation-world/national/article24750439.html. 

For thought-provoking reflections on how people can be incentivized to create outcomes that 
nobody wants, | recommend "Meditations on Moloch,” http:// slatestarcodex.com/2014/07/30/ 
meditations-on-moloch. 
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For an interactive timeline of close calls when nuclear war might have started by accident, 
see Future of Life Institute, “Accidental Nuclear War: A Timeline of Close Calls,” online at 
http:/tinyurl.com/nukeoops. 

For compensation payments made to U.S. nuclear testing victims, see U.S. Department of Justice 
website, “Awards to Date 4/24/2015," at https:/www .justice.gov/civil/awards-date-04242015. 


Report of the Commission to Assess the Threat to the United States from Electromagnetic 
Pulse (EMP) Attack, April 2008, available online at http://www .empcommission.org/docs/ 
A2473-EMP_Commission-7MB.pdf. 


. Independent research by both U.S. and Soviet scientists alerted Reagan and Gorbachev to 


the risk of nuclear winter: P. J. Crutzen and J. W. Birks, “TheAtmosphere After a Nuclear War: 
Twilight at Noon,” Ambio 11, no. 2/3 (1982): 114-125. R. P. Turco, O. B. Toon, T. P. Ackerman, J. 
B. Pollack and C. Sagan, “Nuclear Winter: Global Consequences of Multiple Nuclear Explosions,” 
Science 222 (1983): 1283-1292. V. V. Aleksandrov and G. L. Stenchikov, “On the Modeling of 
the Climatic Consequences of the Nuclear War,” Proceeding on Applied Mathematics (Moscow: 
Computing Centre of the USSR Academy of Sciences, 1983), 21. A. Robock, “Snow and Ice 
Feedbacks Prolong Effects of Nuclear Winter,” Nature 310 (1984): 667-670. 


. Calculation of climate effects of global nuclear war: A. Robock, L. Oman and L. Stenchikov, 


“Nuclear Winter Revisited with a Modern Climate Model and Current Nuclear Arsenals: 
Still Catastrophic Consequences,” Journal of Geophysical Research 12 (2007): D13107. 


usslul Jadll 
For more information, see Anders Sandberg, “Dyson Sphere FAQ," at http:// www.aleph. 
se/nada/dysonFAQ.html. 


Freeman Dyson's seminal paper on his eponymous spheres: Freeman Dyson, “Search 
for Artificial Stellar Sources of Infrared Radiation,” Science, vol. 131 (1959): 1667-1668. 


Louis Crane and Shawn Westmoreland explain their proposed black hole engine in “Are 
Black Hole Starships Possible?,” at http://arxiv.org/pdf/0908.1803.pdf. 

For a nice infographic from CERN summarizing known elementary particles, see http:// 
tinyurl.com/cernparticle. 


This unique video of a non-nuclear Orion prototype illustrates the idea of nuclear-bomb- 
powered rocket propulsion: https:/www.youtube.com/watch?v =E3Lxx2VAYi8. 

Here's a pedagogical introduction to laser sailing: Robert L. Forward, “Roundirip Interstellar 
Travel Using Laser-Pushed Lightsails,” Journal of Spacecraft and Rockets 21, no. 2 
(March-April 1984), available online at http://www.lunarsail .com/LightSail/rit-1.pdf. 


Jay Olson analyzes cosmically expanding civilizations in “Homogeneous Cosmology with 
Aggressively Expanding Civilizations,” Classical and Quantum Gravity 32 (2015), available 
online at http://arxiv.org/abs/1411.4359. 

The first thorough scientific analysis of our far future: Freeman J. Dyson, “Time Without 
End: Physics and Biology in an Open Universe,” Reviews of Modern Physics 51, no. 3 
(1979): 447, available online at http://blog.regehr.org/ extra_files/dyson.pdf. 

Seth Lloyd's above-mentioned formula told us that performing a computational operation 
during a time interval + costs an energy Ezh/4x, where h is Planck's constant. If we want 
to get N operations done one after the other (in series) during a time T, then + = T/N, so 
E/N=hN/4T, which tells us that we can perform N< VET/h serial operations using energy 
E and time T. So both energy and time are resources that it helps having lots of. If you 
split your energy between n different parallel computations, they can run more slowly and 
efficiently, giving N<2 VETn/h. Nick Bostrom estimates that simulating a 100-year human 
life requires about N = 1027 operations. 
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10. If you want to see a careful argument for why the origin of life may require a very rare fluke, 


11. 


placing our nearest neighbors over 10'°%* meters away, | recommend this video by Princeton 
physicist and astrobiologist Edwin Turner: “improbable Life: An Unappealing but Plausible 
Scenario for Life's Origin on Earth,” at https:/www.youtube.com/watch?v=Bt6n6Tu1beg. 


Essay by Martin Rees on the search for extraterrestrial intelligence: https://www .edge. 
org/annual-question/2016/response/26665. 


glull Jaill 
A popular discussion of Jeremy England's work on “dissipation-driven adaptation” can be 
found in Natalie Wolchover, “A New Physics Theory of Life,” Scientific American (January 
28, 2014), available online at https://www.scienti ficamerican.com/article/a-new-physics- 
theory-of-life/. llya Prigogine and Isabelle Stengers's Order Out of Chaos: Man's New 
Dialogue with Nature (New York: Bantam, 1984) lays many of the foundations for this. 
For more on feelings and their physiological roots: William James, Principles of Psychology 
(New York: Henry Holt & Co., 1890), Robert Ornstein, Evolution of Consciousness: The 
Origins of the Way We Think (New York: Simon & Schuster, 1992); Antonio Damäsio, 
Descartes’ Error: Emotion, Reason, and the Human Brain (New York: Penguin, 2005); 
and Anténio Damäsio, Self Comes to Mind: Constructing the Conscious Brain (New York: 
Vintage, 2012). 
Eliezer Yudkowsky has discussed aligning the goals of friendly AI not with our present goals, 
but with our coherent extrapolated volition (CEV). Loosely speaking this is defined as what 
an idealized version of us would want if we knew more, thought faster and were more the 
people we wished we were. Yudkowsky began criticizing CEV shortly after publishing it in 
2004 (http://intelligence.org/files/ CEV.pdf), both for being hard to implement and because 
it's unclear whether it would converge to anything well-defined. 
In the inverse reinforcement-learning approach, a core idea is that the Al is trying to 
maximize not its own goal-satisfaction, but that of its human owner. It therefore has 
incentive to be cautious when it's unclear about what its owner wants, and to do its best 
to find out. It should also be fine with its owner switching it off, since that would imply that 
it had misunderstood what its owner really wanted. 


Steve Omohundro's paper on Al goal emergence, “The Basic AI Drives,” can be found online 
at http:/inyurl.com/omohundro2008. Originally published in Artificial General Intelligence 
2008: Proceedings of the First AGI Conference, ed. Pei Wang, Ben Goertzel and Stan 
Franklin (Amsterdam: IOS, 2008), 483-492. 


A thought-provoking and controversial book on what happens when intelligence is used 
to blindly obey orders without questioning their ethical basis: Hannah Arendi, Eichmann in 
Jerusalem: À Report on the Banality of Evil (New York: Penguin, 1963). A related dilemma 
applies to a recent proposal by Eric Drexler (http://www.fhi.ox.ac.uk/reports/2015-3.pdf) to 
keep superintelligence under control by compartmentalizing it into simple pieces, none of 
which understand the whole picture. If this works, this could again provide an incredibly 
powerful tool without an intrinsic moral compass, implementing its owner's every whim 
without any moral qualms. This would be reminiscent of a compartmentalized bureaucracy 
in a dystopian dictatorship: one part builds weapons without knowing how they'll be used, 
another executes prisoners without knowing why they were convicted, and so on. 

A modern variant of the Golden Rule is John Ravwils' idea that a hypothetical situation is 
fair if nobody would change it without knowing in advance which person in it they'd be. 
For example, the I1Qs of many of Hitler's top officials were found to be quite high. See 
“How Accurate Were the 1Q Scores of the High-Ranking Third Reich Officials Tried at 
Nuremberg?” Quora, available online at http:/tinyurl .com/nurembergiq. 
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10. 


L 


_ 


12. 


13. 


14. 


15. 


galil Jaall 
The entry on consciousness by Stuart Sutherland is quite amusing: Macmillan Dictionary 
of Psychology (London: Macmillan, 1989). 

Erwin Schrôdinger, one of the founding fathers of quantum mechanics, made this remark in 
his book Mind and Matter while contemplating the past—and what would have happened 
if conscious life never evolved in the first place. On the other hand, the rise of Al raises 
the logical possibility that we may end up with a play for empty benches in the future. 


The Stanford Encyclopedia of Philosophy gives an extensive survey of different definitions 
and uses of the word “consciousness”: http:/tinyurl.com/stanford consciousness. 

Yuval Noah Harari, Homo Deus: À Brief History of Tomorrow (New York: HarperCollins, 
2017): 116. 

This is an excellent introduction to System 1 and System 2 from a pioneer: Daniel Kahneman, 
Thinking, Fast and Slow (New York: Farrar, Straus & Giroux, 2011). 


See Christof Koch, The Quest for Consciousness: A Neurobiological Approach (New York: 
W. H. Freeman, 2004). 

Perhaps we're only aware of a tiny fraction (say 10-50 bits) of the information that enters 
our brain each second: K. Küpfmäüller, 1962, “Nachrichtenverarbeitung im Menschen,” 
in Taschenbuch der Nachrichtenverarbeitung, ed. K.Steinbuch (Berlin: Springer-Verlag, 
1962): 1481-1502. T. Norretranders, The User Illusion: Cutting Consciousness Down to 
Size (New York: Viking, 1991). 

Michio Kaku, The Future of the Mind: The Scientific Quest to Understand, Enhance, and 
Empower the Mind (New York: Doubleday, 2014); Jeff Hawkins and Sandra Blakeslee, On 
intelligence (New York: Times Books, 2007); Stanislas Dehaene, Michel Kerszberg and 
Jean-Pierre Changeux, “A Neuronal Model of a Global Workspace in Effortful Cognitive 
Tasks," Proceedings of the National Academy of Sciences 95 (1998): 14529-14534. 
Video celebrating Penfield's famous “! can smell burnt toast” experiment: https: /Avww.youtube. 
com/watch?v=mSN86kphL68. Sensorimotor cortex details: Elaine Marieb and Katja Hoehn, 
Anatomy & Physiology, 3rd ed. (Upper Saddle River, NJ: Pearson, 2008), 391-395. 


The study of neural correlates of consciousness (NCCSs) has become quite mainstream in 
the neuroscience community in recent years —see, e.g., Geraint Rees, Gabriel Kreiman, 
and Christof Koch, “Neural Correlates of Consciousness in Humans,” Nature Reviews 
Neuroscience 3 (2002): 261-270, and Thomas Metzinger, Neural Correlates of Consciousness: 
Empirical and Conceptual Questions (Cambridge, MA: MIT Press, 2000). 


. How continuous flash suppression works: Christof Koch, The Quest for Consciousness: A 


Neurobiological Approach (New York: W. H. Freeman, 2004); Christof Koch and Naotsugu 
Tsuchiya, “Continuous Flash Suppression Reduces Negative Afterimages,” Nature 
Neuroscience 8 (2005): 1096-1101. 

Christof Koch, Marcello Massimini, Melanie Boly and Giulio Tononi, “Neural Correlates of 
Consciousness: Progress and Problems,” Nature Reviews Neuroscience 17 (2016): 307. 


See Koch, The Quest forConsciousness, p. 260, and further discussion in the Stanford 
Encyclopedia of Philosophy, http://tinyurl.com/consciousnessdelay. 

On synchronization of conscious perception: David Eagleman, The Brain: The Story of 
You (New York: Pantheon, 2015), and Stanford Encyclopedia of Philosophy, http:/Ainyurl. 
com/consciousnesssync. 

Benjamin Libet, Mind Time: The Temporal Factor in Consciousness (Cambridge, MA: 
Harvard University Press, 2004); Chun Siong Soon, Marcel Brass, HansJochen Heinze 
and John-Dylan Haynes, “Unconscious Determinants of Free Decisions in the Human 
Brain,” Nature Neuroscience 11 (2008): 543-545, online at http://www.nature.com/neuro/ 
journal/v11/n5/full/nn.2112.html. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 
23. 


24. 


25. 
26. 
27. 


28. 


29. 


30. 
31. 
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Examples of recent theoretical approaches to consciousness: 
Daniel Dennett, Consciousness Explained (Back Bay Books, 1992) 
Bernard Baars, /n the Theater of Consciousness: The Workspace of the Mind (New York: 
Oxford University Press, 2001) 
Christof Koch, The Quest for Consciousness: À Neurobiological Approach (New York: W. 
H. Freeman, 2004) 
Gerald Edelman and Giulio Tononi, À Universe of Consciousness: How Matter Becomes 
Imagination (New York: Hachette, 2008) 
Antonio Damäsio, Self Comes to Mind: Constructing the Conscious Brain (New York: 
Vintage, 2012) 
Stanislas Dehaene, Consciousness and the Brain: Deciphering How the Brain Codes Our 
Thoughts (New York: Viking, 2014) 
Stanislas Dehaene, Michel Kerszberg and Jean-Pierre Changeux, “A Neuronal Model of 
a Global Workspace in Effortful Cognitive Tasks," Proceedings of the National Academy of 
Sciences 95 (1998): 14529-14534 
Stanislas Dehaene, Lucie Charles, Jean-Rémi King and Sébastien Marti, “Toward a Computational 
Theory of Conscious Processing,” Current Opinion in Neurobiology 25 (2014): 760-784 
Thorough discussion of different uses of the term “emergence" in physics and philosophy 
by David Chalmers: http://cse3521.artifice.cc/Chalmers -Emergence.pdf. 
Me arguing that consciousness is the way information feels when being processeld in certain 
complex ways: https://arxiv.org/abs/physics/0510188, https:// arxiv.org/abs/0704.0646, Max 
Tegmark, Our Mathematical Universe (New York: Knopf, 2014). David Chalmers expresses 
a related sentiment in his 1996 book The Conscious Mind: “Experience is information from 
the inside; physics is information from the outside." 
Adenauer Casali et al., “A Theoretically Based Index of Consciousness Independent of 
Sensory Processing and Behavior,” Science Translational Medicine 5 (2013): 198ra105, 
online at http://tinyurl.com/zapzip. 
Integrated information theory doesn't work for continuous systems: - https://arxiv.org/ 
abs/1401.1219 - http:/fjournal.frontiersin.org/article/10.3389/fpsyg.2014.00063/full - https:// 
arxiv.org/abs/1601.02626 
Interview with Clive Wearing, whose short-term memory is only about 30 seconds: https:// 
www.youtube.com/watch?v=WmzU47i2xgw. 
Scott Aaronson IIT critique: http://www.scottaaronson.com/blog/?p=1799. 
Cerrullo IIT critique, arguing that integration isn't a sufficient condition for consciousness: 
http:/tinyurl.com/cerrullocritique. 
IT prediction that simulated humans will be zombies: http://rstb.royalsociety publishing. 
org/content/370/1668/20140167. 
Shanahan critique of IT: https://arxiv.org/pdf/1504.05696.pdf. 
Blindsight: http:/tinyurl.com/blindsight-paper. 
Perhaps we’re only aware of a tiny fraction (say 10-50 bits) of the information that enters our brain 
each second: Küpfmüller, “Nachrichtenverarbeitung im Menschen”; Norretranders, The User {llusion. 
The case for and against “consciousness without access”: Victor Lamme, “How Neuroscience 
Will Change Our View on Consciousness,” Cognitive Neuroscience (2010): 204-220, online 
at http:/{www.tandfoniine.com/doi/abs/10 .1080/17588921003731586. 
“Selective Attention Test,” at https://www.youtube.com/watch?v=vJG69B8U2 Mvo. 
See Lamme, “How Neuroscience Will Change Our View on Consciousness,” n. 28. 
This and other related issues are discussed in detail in Daniel Dennett's book Consciousness 
Explained. 
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32. 
33. 


34. 


35. 


36. 


10. 


11. 


See Kahneman, Thinking, Fast and Slow, cited in n.5. 


The Stanford Encyclopedia of Philosophy reviews the free will controversy: https:// plato. 
stanford.edu/entries/freewill. 


Video of Seth Lloyd explaining why an Al will feel like it has free will: https:// www. youtube. 
com/watch?v=Epj3DF8jDWk. 

See Steven Weinberg, Dreams of a Final Theory: The Search for the Fundamental Laws 
of Nature (New York: Pantheon, 1992). 


The first thorough scientific analysis of our far future: Freeman J. Dyson, “Time Without 
End: Physics and Biology in an Open Universe," Reviews of Modern Physics 51, no. 3 
(1979): 447, available online at http://blog.regehr.org/ extra_files/dyson.pdf. 


daily 

The open letter (htip://futureoflife.org/ai-open-letter) that emerged from the Puerto Rico 
conference argued that research on how to make Al systems robust and benetficial is 
both important and timely, and that there are concrete research directions that can be 
pursued today, as exemplified in this research-priorities document: http://futureoflife.org/ 
data/documents/research_priorities.pdf. 

My video interview with Elon Musk about AI safety can be found on YouTube at https:// 
www.youtube.com/watch?v=rBw0eoZT Y-g. 

Here's a nice video compilation of almost all SpaceX rocket landing attempts, culminating 
With the first successful ocean landing: htips://www.youtube.com/ watch?v=AllaFzIPaGd. 


Elon Musk tweets about our Al-safety grant competition: https://twitter.com/ elonmusk/ 
status/555743387056226304. 


Elon Musk tweets about our Al-safety-endorsing open letter: https://twitter .com/elonmusk/ 
status/554320532133650432. 


Erik Sofge in “An Open Letter to Everyone Tricked into Fearing Artificial Intelligence” 
(Popular Science, January 14, 2015) pokes fun at the scaremongering news coverage of 
our open letter: http://www.popsci.com/open-letter -everyone-tricked-fearing-ai. 

Elon Musk tweets about his big donation to the Future of Life Institute and the world of 
Al-safety researchers: https://twitter.com/elonmusk/status/55574338 7056226304. 


For more about the Partnership on Al to benefit people and society, see their website: 
https://www.partnershiponai.org. 

Some examples of recent reports expressing opinions about Al: One Hundred Year Study 
on Artificial Intelligence, Report of the 2015 Study Panel, "Artificial Intelligence and Life in 
2030" (September 2016), at http:/tinyurl.com/ stanfordai; White House report on the future 
of Al: http///tinyurl.com/obama Alreport; White House report on Al and jobs: htip://tinyurl. 
com/Aljobsreport; IEEE report on AI and human well-being, “Ethically Aligned Design, 
Version1" (December 13, 2016), at http://standards.ieee.org/develop/indconn/ec/ead_v1 
.pdf; road map for U.S. Robotics: http:/tinyurl.com/roboticsmap. 

Among the principles that didn't make the final cut, one of my favorites was this one: 
“Consciousness caution: There being no consensus, we should avoid strong assumptions 
as to whether or not advanced Al would possess or require consciousness or feelings.” It 
went through many iterations, and in the last one, the controversial word “consciousness” 
was replaced by “subjective experience”— but this principle nonetheless got only 88% 
approval, just barely falling short of the 90% cutoff. 

Discussion panel on superintelligence with Elon Musk and other great minds: http://tinyurl. 
com/asilomarAI. 
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